Description

From recent customer issues, we found that our gcsfuse settings are outdated, resulted in poor perf when the dataset has many shards. Details: b/379952790

Tests

Same large dataset, Grain input pipeline
old script:
STANDALONE DATALOADER : First step completed in 41 seconds, on host 0
STANDALONE DATALOADER : 100 batches loaded in 650 seconds, on host 0

New script:
STANDALONE DATALOADER : First step completed in 9 seconds, on host 0
STANDALONE DATALOADER : 100 batches loaded in 21 seconds, on host 0

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed.